.\"
.\" Copyright (C) 1994-2021 Altair Engineering, Inc.
.\" For more information, contact Altair at www.altair.com.
.\"
.\" This file is part of both the OpenPBS software ("OpenPBS")
.\" and the PBS Professional ("PBS Pro") software.
.\"
.\" Open Source License Information:
.\"
.\" OpenPBS is free software. You can redistribute it and/or modify it under
.\" the terms of the GNU Affero General Public License as published by the
.\" Free Software Foundation, either version 3 of the License, or (at your
.\" option) any later version.
.\"
.\" OpenPBS is distributed in the hope that it will be useful, but WITHOUT
.\" ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
.\" FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public
.\" License for more details.
.\"
.\" You should have received a copy of the GNU Affero General Public License
.\" along with this program.  If not, see <http://www.gnu.org/licenses/>.
.\"
.\" Commercial License Information:
.\"
.\" PBS Pro is commercially licensed software that shares a common core with
.\" the OpenPBS software.  For a copy of the commercial license terms and
.\" conditions, go to: (http://www.pbspro.com/agreement.html) or contact the
.\" Altair Legal Department.
.\"
.\" Altair's dual-license business model allows companies, individuals, and
.\" organizations to create proprietary derivative works of OpenPBS and
.\" distribute them - whether embedded or bundled with other software -
.\" under a commercial license agreement.
.\"
.\" Use of Altair's trademarks, including but not limited to "PBS™",
.\" "OpenPBS®", "PBS Professional®", and "PBS Pro™" and Altair's logos is
.\" subject to Altair's trademark licensing policies.
.\"
.TH qsub 1B "25 January 2021" Local "PBS Professional"
.SH NAME
.B qsub 
\- submit a job to PBS 


.SH SYNOPSIS
.B qsub 
[-a <date and time>] [-A <account string>] [-c <checkpoint spec>] 
.RS 5
[-C <directive prefix>] [-e <path>] [-f] [-h] 
.br
[-I [-G [-- <GUI application/script>]] | [-X]] [-j <join>] 
.br
[-J <range> [%<max subjobs]] [-k <discard>] [-l <resource list>] 
.br
[-m <mail events>] [-M <user list>] [-N <name>] [-o <path>]
.br
[-p <priority>] [-P <project>] [-q <destination>] [-r <y|n>]
.br
[-R <remove options>] [-S <path list>] [-u <user list>] 
.br
[-v <variable list>] [-V] [-W <additional attributes>] [-z] 
.br
[- | <script> | -- <executable> [<arguments to executable>]]
.RE
.B qsub
--version

.SH DESCRIPTION
You use the 
.B qsub 
command to submit a batch job to PBS.
Submitting a PBS job specifies a task, requests resources, and sets job attributes.

The 
.B qsub 
command can read from a job script, from standard input, or from the command line.
When the user has submitted the job, PBS returns the job identifier for that job.
For a job, this is of the form:
.RS 3
.I <sequence number>.<server name>
.RE

For an array job, this is of the form:
.RS 3
.I <sequence number>[].<server name>
.RE

During execution, jobs can be interactive or non-interactive.
Interactive jobs are not rerunnable, and if they are blocking, you
cannot use their exit status.

Jobs are run as the user and group who submitted the job.

.B Background Process
.br
By default, on the first invocation, qsub spawns a background process
to manage communication with the PBS server.  Later invocations of
qsub attempt to communicate with this background process.  Under
certain circumstances, calls to qsub when it uses the background
process can result in communication problems.  You can prevent qsub
from spawning a background process by using the 
.I -f 
option, although this can degrade performance.

.B Where PBS Puts Job Files
.br
By default, PBS copies the stdout and stderr files from the job back to the 
current working directory where the 
.B qsub
command is executed.  However, you can specify the output paths using
the 
.I -o 
and 
.I -e 
options.  You can also specify which and whether these
files should be kept on the execution host via the 
.I -k 
option, or deleted, using the 
.I -R 
option.  See the 
.I -k, -o, -e, 
and 
.I -R 
options.

.B Submitting Jobs By Using Scripts
.br
To submit a PBS job by using a script, you specify a job script on the
command line:
.br
.I \ \ \ qsub [<options>] <script name>

For example:
.br
.B \ \ \ qsub myscript.sh

Job scripts are run as the user and group who submitted the job.  Job
scripts can be written in Python, Linux shells such as csh and sh, the
Windows command batch language, Perl, etc.

A PBS job script consists of the following:
.RS 3
Optional shell specification 

Any PBS directives

The user's tasks: programs, commands, or applications 

Optional comments (Under Windows, comments can contain only ASCII
characters.)
.RE

.B Using Shells and Interpreters
.br
By default, PBS uses your login shell to run your script.  You can
optionally specify a different shell or interpreter to run your
script:

Via the 
.I -S 
option to qsub:
.RS 3
.I qsub -S <path to shell> <script name>

For example:
.br
.B qsub -S /bin/bash myscript.sh
.RE

You can specify a different interpreter in the first line of your script.
For example:
.RS 3
.nf
.B cat myscript.sh
#!/bin/bash
#PBS -N MyHelloJob
echo "Hello"
.fi
.RE

.B Python Job Scripts
.br
You can use the same Python script under Linux or under Windows, if
the script is written to be portable.  PBS includes a Python package,
allowing Python job scripts to run; you do not need to install Python.
You can include PBS directives in a Python job script as you would in
a Linux shell script.  Python job scripts can access Win32 APIs,
including the following modules:
.RS 3
Win32api

Win32con

Pywintypes
.RE

For example, we have a Python job script that includes PBS directives:
.RS 3
.B cat myjob.py
.nf
#!/usr/bin/python
#PBS -l select=1:ncpus=3:mem=1gb
#PBS -N HelloJob
print "Hello"
.fi
.RE

So long as the first line of the script is the "#!/usr/bin/python" line
or similar, you don't need to do anything special to run a python script.
.RS 3
.I qsub <script name>
.RE

To run a Python job script under Windows, use the path to the pbs_python
executable on the execution host:
.RS 3
.I qsub -S <pbs_python path on execution host> <script name>

For example, 
.br
qsub -S %PBS_EXEC%\\bin\\pbs_python.exe <script name>
.RE

If the script pathname contains spaces, it must be quoted, for example:
.RS 3
qsub -S "C:\\Program Files\\PBS Pro\\bin\\pbs_python.exe" <script name>
.RE

.B Linux Shell Job Scripts
.br
For example, we have a Linux job script named "weatherscript" for a job named
"Weather1" which runs the executable "weathersim" on Linux:
.RS 3
.nf
#!/bin/sh
#PBS -N Weather1
#PBS -l walltime=1:00:00
/usr/local/weathersim
.RE
.fi

To submit the job, the user types:
.RS 3
.B qsub weatherscript <return>
.RE

.B Windows Command Job Scripts
.br
For example, we have a script named "weather.exe" for a job named "Weather1" which
runs under Windows:
.RS 3
.nf
#PBS -N Weather1
#PBS -l walltime=1:00:00
weathersim.exe
.fi
.RE

To submit the job, the user types:
.RS 3
.B qsub weather.exe <return>
.RE

In Windows, if you use notepad to create a job script, the last line
does not automatically get newline-terminated. Be sure to put one
explicitly, otherwise, PBS job will get the following error message:

   More?

when the Windows command interpreter tries to execute that last line.

.B Submitting Jobs From Standard Input
.br
To submit a PBS job by typing job specifications at the command line,
you type:
.br
.I qsub [<options>] [-] <return>
.br
then type any directives, then any tasks, followed by:
.RS 3
Linux: CTRL-D on a line by itself

Windows: CTRL-Z <return>
.RE

to terminate the input.

The qsub command behaves the same both with and without the dash operand.
.br
For example, on Linux:
.RS 3
.nf
.B qsub <return>
#PBS -N StdInJob
sleep 100
<CTRL-D>
.RE
.fi

.B Submitting Job Directly by Specifying Executable on Command Line
.br
To submit a job directly, you specify the executable on the command
line:
.RS 3
.I qsub [<options>] -- <executable> [<arguments to executable>] <return>
.RE

When you run qsub this way, it runs the executable directly.  It does
not start a shell, so no shell initialization scripts are run, and
execution paths and other environment variables are not set.  There is
not an easy way to run your command in a different directory.  You
should make sure that environment variables are set correctly, and you
will usually have to specify the full path to the command.

For example, to run myprog with the arguments a and b:
.RS 3
.B qsub -- myprog a b <return>
.RE

For example, to run myprog with the arguments a and b, naming the job "JobA":
.RS 3
.B qsub -N JobA -- myprog a b <return>
.RE

On Linux, you need to specify the path to myprog, so the previous example
becomes:
.RS 3
.B qsub -N JobA -- /path/to/myprog a b <return>
.RE

.B Requesting Resources and Placing Jobs
.br
Requesting resources includes setting limits 
on resource usage and controlling how the job is placed on vnodes.

Resources are requested by using the 
.I -l
option, either in job-wide requests using
.I <resource name>=<value>
pairs, or in
.I chunks 
inside of 
.I selection statements.
  See the 
.B pbs_resources(7B) 
man page.

Job-wide 
.I <resource name>=<value> 
requests are of the form:
.RS 3
.I -l <resource name>=<value>[,<resource name>=<value> ...]
.RE

The 
.I selection statement 
is of the form:
.RS 3
.I -l select=[<N>:]<chunk>[+[<N>:]<chunk> ...]
.RE
where 
.I N
specifies how many of that chunk, and a 
.I chunk 
is of the form:
.RS 3
.I <resource name>=<value>[:<resource name>=<value> ...]
.RE

You choose how your chunks are placed using the 
.I place statement.  
The place statement can contain the following elements, in any order:
.RS 3
.I -l place=[<arrangement>][:<sharing>][:<grouping>]

where
.br
.I arrangement 
.RS 3
Whether this chunk is willing to share this vnode or host with other
chunks from the same job.  
.br
One of 
.I free | pack | scatter | vscatter
.RE

.I sharing
.RS 3
Whether this this chunk is willing to share this vnode or host with
other jobs.  
.br
One of 
.I excl | shared | exclhost
.RE

.I grouping 
.RS 3
Whether the chunks from this job should be placed on vnodes that all
have the same value for a resource.  
.br
Can have only one instance of
.I group=<resource name>
.RE

.I free
.RS 3
Place job on any vnode(s).
.RE

.I pack
.RS 3
All chunks are taken from one host.
.RE

.I scatter
.RS 3
Only one chunk with any MPI processes is taken from a host.  A chunk
with no MPI processes may be taken from the same vnode as another
chunk.
.RE

.I vscatter
.RS 3
Only one chunk is taken from any vnode.  Each chunk must fit on a vnode.
.RE

.I excl
.RS 3
Only this job uses the vnodes chosen.
.RE

.I shared
.RS 3
This job can share the vnodes chosen.
.RE

.I exclhost
.RS 3
The entire host is allocated to the job.
.RE

.I group=<resource name>
.RS 3
Chunks are grouped according to a resource.  All vnodes in the group
must have a common value for resource, which can be either the
built-in resource host or a custom vnode-level resource.

.I resource name 
must be a string or a string array.
.RE

The place statement cannot begin with a colon.  Colons are delimiters;
use them only to separate parts of a place statement, unless they are
quoted inside resource values.
.RE

Note that vnodes can have 
.I sharing 
attributes that override job placement requests.  
See the 
.I pbs_node_attributes.7B
man page.

.B Caveats for Requesting Resources
.br
Do not mix old-style resource or vnode specifications with the new
select and place statements.  Do not use one in a job script and the
other on the command line.  Mixing the two will result in an error.

You cannot submit a job requesting a custom resource which has been
created to be invisible or read-only for unprivileged users,
regardless of your privilege.  A Manager or Operator can use the
.B qalter 
command to change a job's request for this kind of custom resource.

For more on resource requests, usage limits and job placement, see
.I pbs_resources(7B).



.B Setting attributes
.br
The job submitter sets job attributes by giving options to the 
.B qsub 
command or by using 
.I PBS directives.
Most qsub options set a job attribute, and have a corresponding PBS
directive with the same syntax as the option.  Attributes set via
command-line options take precedence over those set using PBS
directives. See
.I pbs_job_attributes.7B.

.B Changing qsub Behavior
.br
The behavior of the 
.B qsub
command
may be affected by 
the server's 
.I default_qsub_arguments
attribute.  
This attribute can set the default for any job attribute.  
The 
.I default_qsub_arguments
server attribute is settable by the administrator,
and is overridden by command-line arguments and
script directives.
See the 
.I pbs_server_attributes(1B) 
man page.

The behavior of the 
.B qsub 
command may also be affected by 
any site hooks.  Site hooks can modify the job's attributes, 
change its routing, etc.

.SH Options to qsub

.IP "-a <date and time>" 8
Point in time after which the job is eligible for execution.
Given in pairs of digits.  Sets job's 
.I Execution_Time
attribute to 
.I date and time.
Format: 
.I datetime, 
expressed as
.RS 11
.I [[[[CC]YY]MM]DD]hhmm[.SS]
.RE
.IP " " 8
where 
.I CC 
is the century,
.I YY 
is the year, 
.I MM 
is the month,
.I DD 
is the day of the month, 
.I hh 
is the hour, 
.I mm 
is the minute, and 
.I SS 
is the seconds.

.IP " " 8
Each portion of the date defaults to the current date, as long as the 
next-smaller portion is in the future.  For example, if today is the
3rd of the month and the specified day 
.I DD 
is the 5th, the month 
.I MM
is set to the current month.

If a specified portion has already passed, the next-larger portion is set
to one after the current date.  For example, if the day
.I DD
is not specified, but the hour 
.I hh 
is specified to be 10:00 a.m. and the current time is 11:00 a.m., 
the day 
.I DD
is set to tomorrow.

.IP "-A <account string>" 8
Accounting string associated with the job.  Used for labeling accounting data.
Sets job's 
.I Account_Name 
attribute to 
.I account string.
.br
Format: 
.I String

.IP "-c <checkpoint spec>"
Determines when the job will be checkpointed.  Sets job's 
.I Checkpoint
attribute to 
.I checkpoint spec.  
An 
.I $action
script is required to checkpoint the job.  

The argument 
.I checkpoint spec
can take on one of the following values:
.RS
.IP c 5
Checkpoint at intervals, measured in CPU time, set on job's execution
queue.  If there is no interval set at the queue, the job is not
checkpointed.

.IP "c=<minutes of CPU time>" 5
Checkpoint at intervals of specified number of minutes of job CPU
time.  This value must be greater than zero.  If the interval
specified is less than that set on the job's execution queue, the
queue's interval is used.
.br
Format: 
.I Integer
.IP w 5
Checkpoint at intervals, measured in walltime, set on job's execution
queue.  If there is no interval set at the queue, the job is not
checkpointed.

.IP "w=<minutes of walltime>" 5
Checkpoint at intervals of the specified number of minutes of job walltime.
This value must be greater
than zero.  If the interval specified is less than that set on the
job's execution queue, the queue's interval is
used.
.br
Format: 
.I Integer
.IP n 5
No checkpointing.
.IP s 5
Checkpoint only when the server is shut down.
.IP u 5
Unset.  Defaults to behavior when 
.I interval
argument is set to 
.I s.
.LP
Default: 
.I u
.br
Format: 
.I String
.RE
.RE

.IP "-C <directive prefix>" 8
Defines the prefix identifying a 
.I PBS directive.
Default prefix is "#PBS".
.IP
If the
.I directive prefix
argument is a null string, qsub
does not scan the script file for directives.
Overrides the PBS_DPREFIX environment variable and the default.
The string "PBS_DPREFIX" cannot be used as a PBS directive.
Length limit: 4096 characters.

.IP "-e <path>" 8
Path to be used for the job's standard error stream.
Sets job's 
.I Error_Path 
attribute to 
.I path.
The
.I path
argument is of the form:
.RS 13
.I [<hostname>:]<path>
.RE
.RS
The 
.I path 
is interpreted as follows:

.IP path 5
If
.I path
is relative, it is taken to be relative to the current working directory of the 
.B qsub
command, where it is executing on the current host.

If
.I path
is absolute, it is taken to be an absolute path on the current host where the 
.B qsub
command is executing.

.IP hostname:path
If 
.I path
is relative, it is taken to be relative to the user's 
home directory on the host named
.I hostname.

If 
.I path
is absolute, it is an absolute path on the host named
.I hostname.
.LP
If 
.I path
does not include a filename, the default filename has the form
.I <job ID>.ER

If the
.I -e
option is not specified, PBS copies the standard error to the current
working directory where the 
.B qsub 
command was executed, and writes standard error to the default filename, 
which has this form:
.br
.I <job name>.e<sequence number>

If you use a UNC path for output or error files, the 
.I hostname 
is optional.  If you use a non-UNC path, the 
.I hostname 
is required.

This option is overridden by the 
.I -k
option.
.RE

.IP "-f" 8
Prevents 
.B qsub
from spawning a background process.  By default, 
.B qsub 
spawns a background process to manage communication with the PBS server.  
When this option is specified, the 
.B qsub
process connects directly to the server and no background process is created.

NOTE: Use of this option degrades performance of the 
.B qsub
command when calls are made in rapid succession.

.IP "-G [-- <GUI application>]" 8
Starts a GUI session.  When no application or script is provided,
starts a GUI-enabled interactive shell.  When an application or script
is provided, starts the GUI application or script.  Use full path to
application or script unless the path is part of the user's
PATH environment variable on the execution host.  When submission and
execution hosts are different, this uses a remote viewer.
.br
Session is terminated when remote viewer, GUI application, or 
interactive shell is terminated, or when job is deleted.
.br
Can be used only with interactive jobs (the 
.I -I 
option).
.br
Available only under Windows.

.IP "-h" 8
Applies a 
.I User 
hold to the job.
Sets the job's 
.I Hold_Types 
attribute to "u".

.IP "-I" 8
Job is to be run interactively.  Sets job's 
.I interactive
attribute to 
.I True.
The job is queued
and scheduled as any PBS batch job, but when executed, the standard input,
output, and error streams of the job are connected to the
terminal session in which 
.B qsub 
is running.
If a job script is given, only its directives are processed.  When the job
begins execution, all input to the job is taken from the terminal session.

Interactive jobs are not rerunnable.

Job arrays cannot be interactive.

When used with 
.I -Wblock=true, 
no exit status is returned.

.IP "-j <join>" 8
Specifies whether and how to join the job's standard error and standard output streams.
Sets job's 
.I Join_Path
attribute to 
.I join.
Default: 
.I n, 
not merged.  
The 
.I join
argument can take the following values:
.RS
.IP oe 11
Standard error and standard output are merged into standard output.

.IP eo 11
Standard error and standard output are merged into standard error.

.IP n 11
Standard error and standard output are not merged.
.RE

.IP "-J <range> [%<max subjobs>]" 8
Makes this job an array job.  Sets job's 
.I array
attribute to 
.I True.

Use the
.I range 
argument to specify the indices of the subjobs of the array.
.I range 
is specified in the form
.I X-Y[:Z]  
where 
.I X 
is the first index, 
.I Y 
is the upper bound on the indices, and
.I Z 
is the stepping factor.  For example,  2-7:2 will produce indices of 2, 4, and
6.  If 
.I Z 
is not specified, it is taken to be 1.  Indices must be greater than or 
equal to zero.

Use the optional 
.I %max subjobs 
argument to set a limit on the number of subjobs that can be running
at one time.  This sets the value of the 
.I max_run_subjobs 
job attribute to the specified maximum.

Job arrays are always rerunnable.

.IP "-k <discard>" 8
Specifies whether and which of the standard output and standard error streams
is left behind on the execution host, or written to their final destination.
Sets the job's 
.I Keep_Files 
attribute to 
.I discard.
Overrides default path names for these streams.  
Overrides 
.I -o
and 
.I -e
options.

Default: 
.I n; 
neither is retained, and files are not written to their final destinations.

In the case where output and/or error is retained on the execution host in
a job-specific staging and execution directory created by PBS, these
files are deleted when PBS deletes the directory.

The 
.I discard
argument can take the following values:
.RS
.IP e 5
The standard error stream is retained on the execution host, in the 
job's staging and execution directory.  The filename is
.RS
.RS 3
.I <job name>.e<sequence number>
.RE
.RE

.IP o  5
The standard output stream is retained on the execution host, in the
job's staging and execution directory.  The filename is
.RS
.RS 3
.I <job name>.o<sequence number>
.RE
.RE

.IP "eo, oe"  5
Both standard output and standard error streams are 
retained on the execution host, in the
job's staging and execution directory. 

.IP d 5
Output and/or error are written directly to their final destination.
Overrides action of leaving files on execution host.

.IP n  5
Neither stream is retained.
.RE

.IP "-l <resource list>" 8
.RS
Allows the user to request resources and specify job placement.  Sets job's 
.I Resource_list 
attribute to 
.I resource list.
Requesting a resource places a limit on its usage.

For how to request resources and place jobs, see 
.B "Requesting Resources and Placing Jobs" above.
.RE

.IP "-m <mail events> " 8
Specifies the set of conditions under which mail about the job is sent.
Sets job's 
.I Mail_Points
attribute to 
.I mail events.  
The 
.I mail events
argument can be one of the following:
.RS 11
The single character "n"
.br
Any combination of "a", "b", and "e", with optional "j"
.RE
.IP
The following table lists the sub-options to the 
.I -m 
option:
.RS
.IP n 5
No mail is sent
.IP a 5
Mail is sent when the job is aborted by the batch system
.IP b 5
Mail is sent when the job begins execution
.IP e 5
Mail is sent when the job terminates
.IP j 5
Mail is sent for subjobs. Must be combined with one or more of the
.I a
, 
.I b
, or 
.I e 
sub-options
.RE
.IP
Format: 
.I String
.br
Syntax: 
.I n | [j](one or more of a, b, e)
.br
Example: -m ja
.br
Default: 
.I "a"  

.IP "-M <user list>" 8
List of users to whom mail about the job is sent.  Sets job's 
.I Mail_Users 
attribute to 
.I user list.  

.RS
The
.I user list
argument has the form:
.RS 3
.I <username>[@<hostname>][,<username>[@<hostname>],...]
.RE

Default: Job owner
.RE

.IP "-N <name> " 8
Sets job's 
.I Job_Name 
attribute and name to 
.I name.  

Format: 
.I Job Name

Default: if a script is used to submit the job, the job's name is the
name of the script.  If no script is used, the job's name is "STDIN".

.IP "-o <path>" 8
Path to be used for the job's standard output stream.
Sets job's 
.I Output_Path 
attribute to 
.I path.
The
.I path
argument has the form:
.RS 13
.I [<hostname>:]<path>
.RE
.RS
The 
.I path 
is interpreted as follows:
.IP path 5
If
.I path 
is relative, it is taken to be relative to 
the current working directory of the
.B qsub
command, where it is executing on the current host.

If
.I path
is absolute, it is taken to be an absolute path on 
the current host where the
.B qsub
command is executing.
.IP hostname:path
If 
.I path
is relative, it is taken to be relative to the user's 
home directory on the host named
.I hostname.

If 
.I path 
is absolute, it is an absolute path on the host named
.I hostname.
.LP

If 
.I path
does not include a filename, the default filename has the form 
.I <job ID>.OU

If the
.I -o
option is not specified, PBS copies the standard output to the current
working directory where the 
.B qsub 
command was executed, and writes standard output to the default filename,
which has this form: 
.I <job name>.o<sequence number>

If you use a UNC path, the hostname is optional.  If you use a non-UNC
path, the hostname is required.

This option is overridden by the 
.I -k
option.
.RE

.IP "-p <priority>" 8
Priority of the job.  Sets job's 
.I Priority
attribute to 
.I priority.

Format: host-dependent integer
.br
Range: 
.I [-1024, +1023] 
inclusive
.br
Default: 
.I Zero  

.IP "-P <project>" 8
Specifies a project for the job. Sets job's 
.I project
attribute to 
.I project.

Format: 
.I Project Name

Default value: "_pbs_project_default"

.IP "-q <destination>" 8
Where the job is sent upon submission.  

Specifies a queue, a server, or a queue at a server.  The destination
argument can have one of these formats:
.RS
.I <queue name>
.RS 3
Job is submitted to the specified queue at the default server.
.RE

.I @<server name>
.RS 3
Job is submitted to the default queue at the specified server.
.RE

.I <queue name>@<server name>
.RS 3
Job is submitted to the specified queue at the specified server.
.RE

Default: Default queue at default server 
.RE

.IP "-r <y|n>" 8
Declares whether the job is rerunnable.  Sets job's 
.I Rerunable
attribute to the argument value.
Does not affect how the job is handled in the case where the job was
unable to begin execution.

Format: Single character, "y" or "n"
.br

.RS
.IP y 5
Job is rerunnable.
.IP n 5
Job is not rerunnable.
.RE

.IP "" 8
Default: "y"

Interactive jobs are not rerunnable.
Job arrays are automatically marked as rerunnable.
See the
.I qrerun.1B
man page.

.IP "-R <remove options>" 8
Specifies whether standard output and/or standard error files are
automatically removed (deleted) upon job completion.

Sets the job's 
.I Remove_Files 
attribute to 
.I remove options.  
Overrides default path names for these streams.  Overrides 
.I -o 
and 
.I -e 
options.

This attribute cannot be altered once the job has begun execution.

Default: 
.I Unset; 
neither is removed

The 
.I remove options 
argument can take the following values:
.RS
.IP e 5     
The standard error stream is removed (deleted) upon job completion
.IP o 5      
The standard output stream is removed (deleted) upon job completion
.IP "eo, oe" 5  
Both standard output and standard error streams are removed (deleted) upon job completion
.IP unset 5  
Neither stream is removed.
.RE

.IP "-S <path list>" 8
Specifies the interpreter or shell path for the job script.  Sets job's 
.I Shell_Path_List 
attribute to 
.I path list.

The 
.I path list
argument is the full path to the interpreter or shell including the 
executable name.  

Only one path may be specified without a host name.
Only one path may be specified per named host.  The path selected
is the one whose host name is that of the server on which the job
resides.  

.RS
Format:
.RS 3
.I <path>[@<hostname>][,<path>@<hostname> ...]
.RE

Default: user's login shell on execution host

Example of using bash via a directive:
.RS 3
.B #PBS -S /bin/bash@mars,/usr/bin/bash@jupiter
.RE
Example of running a Python script from the command line on Linux: 
.RS 3
.B qsub -S $PBS_EXEC/bin/pbs_python <script name>
.RE
Example of running a Python script from the command line on Windows: 
.RS 3
qsub -S %PBS_EXEC%\\bin\\pbs_python.exe <script name>
.RE
.RE
.IP 

.IP "-u <user list>" 8
List of usernames.  Job is run under a username from this list.
Sets job's 
.I User_List 
attribute to 
.I user list.

Only one username may be specified without a hostname.
Only one username may be specified per named host.  
The server on which the job resides will select first the username whose
hostname is the same as the server name.  Failing that, 
the next selection is the username with no specified hostname.
The usernames on the server and execution hosts must be the same.
The job owner must have authorization to run as the specified user.

.RS
Format of
.I user list: 
.RS 3
.I <username>[@<hostname>][,<username>@<hostname> ...]
.RE

Default: job owner (username on submission host)  
.RE

.IP "-v <variable list>"
Lists environment variables and shell functions to be exported to the job.
This is the list of environment variables that are added to
those already automatically exported.  These variables exist in
the user's environment from which
.B qsub
is run.
The job's 
.I Variable_List
attribute is appended with the variables in
.I variable list 
and their values.
See the 
.B ENVIRONMENT
section of this man page.

.RS
Format: comma-separated list of strings in the form:
.RS 3
.I <variable>
.RE
or
.RS 3
.I <variable>=<value>
.RE

If a 
.I <variable>=<value> 
pair contains any commas, the value must be 
enclosed in single or double quotes, and the 
.I <variable>=<value> 
pair must be enclosed in the kind of quotes not used to 
enclose the value.  For example:
.RS 3
qsub -v "var1='A,B,C,D'" job.sh
.br
qsub -v "a=10,var2='A,B',c=20,d='Hello world'" job.sh
.RE

Default: no environment variables are added to the job's variable list.
.RE

.IP "-V" 8
All environment variables and shell functions in the user's
environment where
.B qsub
is run are exported to the job.
The job's
.I Variable_List
attribute is appended with all of these environment variables and
their values.

.IP "-W <additional attributes>" 8
The 
.I -W 
option allows specification of some job attributes.  Some
job attributes must be specified using this option.  Those attributes
are listed below.
.RS
Format:
.RS 3
.I -W <attribute name>=<value>[,<attribute name>=<value>...]
.RE

If white space occurs within the 
.I additional attributes
argument, or the equal sign "=" occurs within a
.I value
string, it must be enclosed with single quotes or double quotes.

The following attributes can be set using the -W option only

.I "block=true"
.RS 3
The 
.B qsub
command waits for the job to terminate, then returns the job's exit value.
Sets job's 
.I block
attribute to 
.I True.
When used with X11 forwarding or interactive jobs, no exit value is returned.
See 
.B EXIT STATUS
section in this man page.
.LP
.RE

.I "create_resv_from_job=<value>"
.RS 3
When this job starts, immediately creates and confirms a job-specific
start reservation on the same resources as the job (including
resources inherited by the job), and places the job in the
job-specific reservation queue.  Sets the job's
.I create_resv_from_job 
attribute to 
.I True.  
Sets the job-specific reservation's 
.I reserve_job 
attribute to the ID of the job from which the reservation was created.
The new reservation's duration and start time are the same as the
job's walltime and start time.  If the job is peer scheduled, the
job-specific reservation is created in the pulling complex.
.br
Format: 
.I Boolean 
.br
Example: 
.B qsub myscript.sh -Wcreate_resv_from_job=1 

Cannot be used with job arrays or jobs being submitted into a reservation.
.RE

.I "depend=<dependency list>"
.RS 3
Defines dependencies between this and other jobs.  
Sets the job's
.I depend
attribute to 
.I dependency list.
The 
.I dependency list
has the form:
.RS 3
.I <type>:<arg list>[,<type>:<arg list> ...]
.RE
where except for the 
.I on
type, the
.I arg list
is one or more PBS job IDs, and has the form:
.RS 3
.I <job ID>[:<job ID> ...]
.RE
The 
.I type 
can be:

.IP "after: <arg list>" 4
This job may be scheduled for execution at any point after all jobs
in 
.I arg list
have started execution.

.IP "afterok: <arg list>" 4
This job may be scheduled for execution only after all jobs in
.I arg list
have terminated with no errors.
See "Warning about exit status with csh" in 
.B EXIT STATUS.

.IP "afternotok: <arg list>" 4
This job may be scheduled for execution only after all jobs in 
.I arg list
have terminated with errors.
See "Warning about exit status with csh" in 
.B EXIT STATUS.

.IP "afterany: <arg list>" 4
This job may be scheduled for execution after all jobs in
.I arg list
have finished execution, with any exit status (with or without errors.)
This job will not run if a job in the 
.I arg list 
was deleted without ever having been run.

.IP  "before: <arg list> " 4
Jobs in 
.I arg list 
may begin execution once this job has begun execution.
It is uncommon for users to specify a before condition. Rather,
PBS adds before dependencies automatically to the targets of
after dependencies.

.IP  "beforeok: <arg list>" 4
Jobs in 
.I arg list
may begin execution once this job terminates without errors.
See "Warning about exit status with csh" in 
.B EXIT STATUS.

.IP  "beforenotok: <arg list>" 4
If this job terminates execution with errors, jobs in 
.I arg list
may begin.
See "Warning about exit status with csh" in 
.B EXIT STATUS.

.IP  "beforeany: <arg list>" 4
Jobs in 
.I arg list
may begin execution once this job terminates execution,
with or without errors.

.IP "on: count" 4
This job may be scheduled for execution after 
.I count 
dependencies on
other jobs have been satisfied.  This type is used in conjunction
with one of the 
.I before
types listed.
.I count 
is an integer greater than 0.
.LP

Job IDs in the
.I arg list 
of 
.I before 
types must have been submitted with a 
.I type 
of 
.I on.

To use the 
.I before types,
the user must have the authority to alter the jobs in 
.I arg list.
Otherwise, the dependency is rejected and the new job aborted.

Error processing of the existence, state, or condition of the job on
which the newly submitted job is performed after the job is queued.
If an error is detected, the new job is deleted by the server.  Mail
is sent to the job submitter stating the error.

Dependency example:
.RS 3
In this example, we save the output (the jobid) from the first qsub into
shell variable "jobid" so we can supply it to the depend option on the
second job.
.LP
.B "jobid=`qsub first_step.sh`"
.br
.B "qsub -W depend=afterok:$jobid second_step.sh"
.br
.RE
.RE

.I "group_list=<group list>"
.RS 3
List of group names.  Job is run under a group name from this list.
Sets job's
.I group_List
attribute to
.I group list.

Only one group name may be specified without a hostname.
Only one group name may be specified per named host.
The server on which the job resides will select first the group name whose
hostname is the same as the server name.  Failing that,
the next selection is the group name with no specified hostname.
The group names on the server and execution hosts must be the same.
Job submitter's primary group is automatically added to this 
list.  
.LP
Under Windows, the primary group is the first group found for 
the user by PBS when it queries the accounts database.

Format of
.I group list:
.RS 3
.I <group name>[@<hostname>][,<group name>@<hostname> ...]
.RE

Default: Login group name of job owner
.RE

.I pwd
.br
.I pwd=''
.br
.I pwd=""
.RS 3
These forms prompt the user for a password.  A space between
.I W
and
.I pwd
is optional.  Spaces between the quotes are optional.
Examples:
.nf
    qsub ... -Wpwd <return>
    qsub ... -W pwd='' <return>
    qsub ... -W pwd="  " <return>
.fi
Available on supported Linux platforms only.
.RE

.I release_nodes_on_stageout=<value>
.RS 3
When set to 
.I True, 
all of the job's vnodes not on the primary execution
host are released when stageout begins.

Cannot be used with vnodes tied to Cray X* series systems.

When cgroups is enabled and this is used with some but not all vnodes
from one MoM, resources on those vnodes that are part of a cgroup are
not released until the entire cgroup is released.

The job's 
.I stageout 
attribute must be set for the
.I release_nodes_on_stageout 
attribute to take effect.

Format: 
.I Boolean

Default: 
.I False
.RE

.I "run_count=<value>"
.RS 3
Sets the number of times the server thinks it has run the job.  Sets the value of
the job's 
.I run_count
attribute to 
.I value.  

Format: Integer greater than or equal to zero
.RE

.I "sandbox=<sandbox spec>"
.RS 3
Determines which directory PBS uses for the job's staging and execution.  Sets
job's 
.I sandbox
attribute to 
.I sandbox spec.

Allowed values for 
.I sandbox spec:

.IP PRIVATE 5
PBS creates a job-specific directory for staging and execution.
.IP "HOME or unset" 5
PBS uses the user's home directory for staging and execution.
.LP

Format:
.I String
.RE

.I "stagein=<path list>"
.br
.I "stageout=<path list>"
.RS 3
Specifies files or directories to be staged in before execution or staged out
after execution is complete.  Sets the job's 
.I stagein
and 
.I stageout
attributes to the specified
.I path lists.
On completion of the job, all staged-in and staged-out files and directories
are removed from the execution host(s).  The
.I path list
has the form:
.RS 3
.I <file spec>[,<file spec>]
.RE
where 
.I file spec 
is 
.RS 3
.I <execution path>@<hostname>:<storage path>
.RE
regardless of the direction of the copy.
The name
.I execution path
is the name of the file or directory on the primary execution host.
It can be relative to the staging and execution directory on the
execution host, or it can be an absolute path.

The "@" character separates 
.I execution path
from 
.I storage path.

The name
.I storage_path
is the path on 
.I <hostname>. 
The storage_path can be absolute, or it can be relative to the user's home
directory on hostname.

If 
.I path list
has more than one 
.I file spec,
i.e. it contains commas, it must be enclosed in double quotes.

If you use a UNC path, the 
.I hostname 
is optional.  If you use a non-UNC path, the 
.I hostname 
is required.
.RE

.I "umask=<mask value>"
.RS 3
The umask with which the job is started.  Sets job's 
.I umask
attribute to 
.I mask value.
Controls umask of job's standard output and standard error.

The following example allows group and world read on the job's output:
.RS 3
.B -W umask=33
.RE

Format: One to four octal digits; typically two

Default value: 
.I 077
.RE
.RE

.IP "-X" 8
Allows user to receive X output from interactive job.

DISPLAY variable in submission environment must be set to 
desired display.

Can be used with interactive jobs only: must be used with
one of the following:
.RS
.RS 3
.I -I 
.br
.I -W interactive=true
.B (deprecated)
.RE

Cannot be used with 
.I -v DISPLAY.

When used with 
.I -Wblock=true,
no exit status is returned.

Can be used with 
.I -V 
option.

Not available under Windows.
.RE

.IP "-z" 8
Job identifier is not written to standard output.
.RE

.IP "--version" 8
The 
.B qsub
command returns its PBS version information and exits.
This option can only be used alone.

.SH  OPERANDS
The 
.B qsub 
command accepts as operands one of the following:
.RS 3
.I (no operands)
.RS 3
Same as with a dash. Any PBS directives and user tasks are read
from the command line.
.RE

.I <script>
.RS 3
Path to script.  Can be absolute or relative to current directory
where 
.B qsub 
is run.
The script must be the last argument to
.BR qsub .
.RE

.I "-"
.RS 3
When you use a dash, any PBS directives and user tasks are read
from the command line.
.RE

.I -- <executable> [<arguments to executable>]
.RS 3
A single executable (preceded by two dashes) and its arguments

The executable, and any arguments to the executable, are given on the
qsub command line.  The executable is preceded by two dashes, "--".
All
.B qsub
options must come before the "--".

When you run qsub this way, it runs the executable directly.  It does
not start a shell, so no shell initialization scripts are run, and
execution paths and other environment variables are not set.  You
should make sure that environment variables are set correctly.
.RE
.RE

.SH STANDARD OUTPUT
.IP "Job ID for submitted jobs" 8
If the job is successfully created

.IP "(No output)" 8
If the 
.I -z
option is set

.SH STANDARD ERROR
The
.B qsub
command writes a diagnostic message to standard error for
each error occurrence.

.SH ENVIRONMENT VARIABLES
The
.B qsub
command uses the following environment variables:
.RS 3
.IP PBS_DEFAULT 3
Name of default server.

.IP PBS_DPREFIX 3
Prefix string which identifies PBS directives.
.RE

PBS automatically exports the following environment variables to the job.
Environment variables beginning with "PBS_O_" are created by 
.B qsub
and included in the job's Variable_List.

.IP PBS_ENVIRONMENT 8
Set to 
.I PBS_BATCH 
for a batch job.  Set to 
.I PBS_INTERACTIVE
for an interactive job.

.IP PBS_JOBDIR 8
Pathname of job's staging and execution directory on the
primary execution host.  

.IP PBS_JOBID 8
Job identifier given by PBS when the job is submitted.

.IP PBS_JOBNAME 8
Job name specified by submitter.

.IP PBS_NODEFILE 8
Name of file containing the list of vnodes assigned to the job when
the job runs.

.IP PBS_O_HOME 8
User's home directory.  
Value of HOME taken from user's submission environment.

.IP PBS_O_HOST 8
Name of submit host.
Value taken from user's submission environment.

.IP PBS_O_LANG 8
Value of LANG taken from user's submission environment.

.IP PBS_O_LOGNAME 8
User's login name.
Value of LOGNAME taken from user's submission environment.

.IP PBS_O_MAIL 8
Value of MAIL taken from user's submission environment.

.IP PBS_O_PATH 8
User's PATH.
Value of PATH taken from user's submission environment.

.IP PBS_O_QUEUE 8
Name of the queue to which the job was submitted.
Value taken from job submission, otherwise default queue.

.IP PBS_O_SHELL 8
Value of SHELL taken from user's submission environment.

.IP PBS_O_SYSTEM 8
Operating system, from 
.I uname -s, 
on submit host.
Value taken from user's submission environment.

.IP PBS_O_TZ 8
Timezone.  Value taken from user's submission environment.

.IP PBS_O_WORKDIR 8
Absolute path to directory where
.B qsub
is run.
Value taken from user's submission environment.

.IP PBS_QUEUE 8
Name of the queue from which the job is executed.

.IP TMPDIR 8
Pathname of job's scratch directory.  Set when PBS assigns it.

.SH EXIT STATUS
For non-blocking jobs:
.RS 3
.IP Zero 8
Upon successful processing of input

.IP "Greater than zero" 8
Upon failure of 
.B qsub
.RE

For blocking jobs:
.RS 3
.IP "Exit value of job" 8
When job runs successfully

.IP 3 8
If the job is deleted without being run
.RE

.B Warning About Exit Status with csh:
.br
If a job is run in csh and a .logout file
exists in the user's home directory on the host where the job executes,
the exit status
of the job is that of the .logout script, not the job script.  This may
impact any inter-job dependencies.  

.SH SEE ALSO
pbs_job_attributes(7B),
pbs_server_attributes(7B),
pbs_resources(7B),
qalter(1B), 
qhold(1B), 
qmove(1B), 
qmsg(1B), 
qrerun(1B),
qrls(1B), 
qselect(1B), 
qstat(1B)

